Image processing device, image processing method, and recording medium

ABSTRACT

An image processing device ( 3000 ) comprises an input unit ( 3020 ) and a presentation unit ( 3040 ). The input unit ( 3020 ) accepts an input of an operation for movement, on a captured image captured by a camera, of a first image which is superimposed on the captured image on the basis of a predetermined camera parameter indicating the position and attitude of the camera and which indicates a target object having a predetermined shape and a predetermined size set in a real space. The presentation unit ( 3040 ) presents the first image indicating the target object in a manner of view corresponding to a position on the captured image after the movement on the basis of the camera parameter.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/131,306, filed Dec. 22, 2020, which is a continuation of U.S. patentapplication Ser. No. 15/512,340, filed Mar. 17, 2017, now U.S. Pat. No.10,911,645, which is a National Stage of International Application No.PCT/JP2015/071750 filed Jul. 31, 2015, which claims priority fromJapanese Patent Application No. 2014-257137, filed on Dec. 19, 2014, andJapanese Patent Application No. 2014-191480, filed on Sep. 19, 2014. Theentire disclosures of the above-referenced applications are incorporatedherein by reference in their entirety.

TECHNICAL FIELD

The present invention relates to an image processing technique.

BACKGROUND ART

As one method for monitoring facilities and the like, there is a methodthat uses a video of a monitoring camera installed in the facilities andthe like. An actual size and position of a person or object appearing ina video of the monitoring camera may be calculated using information(hereinafter, referred to as camera parameters) on a position andattitude (posture) of the camera and a size and position on an image ofthe person or object appearing in the video (image). Through suchcalculation, it is possible to grasp, when, for example, an importantperson (a criminal of a case or the like) is appearing in a video of amonitoring camera, a height and the like of the person using the videoof the monitoring camera.

Camera parameters used in the above-described purpose and the like areestimated, for example, by calibration. NPL 1 discloses a method inwhich a calibration pattern is image-captured by a camera and cameraparameters (a rotation and translation of the camera) indicating aposition and attitude of the camera are estimated from an associationrelation between three-dimensional coordinates of the calibrationpattern in a real world and two-dimensional coordinates of thecalibration pattern of the captured image.

Further, there is a case in which already-estimated camera parametersare acquired and used. For example, camera parameters previouslycalculated by executing calibration for a camera having been a pasttarget may be acquired, or camera parameters defined on the basis ofinformation such as a position and attitude upon installation of thecamera may be acquired.

CITATION LIST Non Patent Literature

NPL 1: Gang Xu and Saburo Tsuji, “Three-dimensional Vision”, KyoritsuShuppan, pp. 79-82, 1998

SUMMARY OF INVENTION Technical Problem

It is difficult for camera parameters to always appropriately indicate aposition and attitude or the like of a camera that is a target. Forexample, in a method for calculating camera parameters by calibration,due to a cause such as an input error of a corresponding point, lensdistortion, and the like, camera parameters indicating a position andattitude different from an actual position and attitude of a camera maybe calculated. Further, also when an already-estimated cameral parameteris acquired, it is difficult to understand whether the camera parametersare appropriate. It is possible that, for example, with an elapsed time,a position and attitude of a camera may change, and therefore cameraparameters estimated in the past and a current position and attitude ofthe camera may differ from each other.

When the camera parameters do not appropriately indicate a position andattitude or the like of a camera that is a target, a problem that anerror in a calculation result occurs upon calculating, for example, aheight of an important person appearing in a video of theabove-described monitoring camera is produced.

In view of the above-described problem, an object of the presentinvention has been achieved. The object of the present invention is toprovide a technique enabling a use to easily confirm whether cameraparameters are appropriate.

Solution to Problem

A first image processing device provided by the present inventionincludes: an input means configured to accept inputting of an operationfor movement, on a captured image captured by a camera, to a first imagethat is superimposed on the captured image on the basis of predeterminedcamera parameters indicating a position and attitude of the camera andindicates a target object having a predetermined shape and apredetermined size set on a real space; and a presentation meansconfigured to present the first image indicating the target object in amanner of view relating to a position on the captured image after themovement on the basis of the camera parameters.

A second image processing device provided by the present inventionincludes: a display means configured to display a captured imagecaptured by a camera; a parameter acquisition means configured toacquire a cameral parameter indicates a position and an attitude of thecamera; an input means configured to accept designation of a firstposition in the captured image; and a presentation means configured to,based on the camera parameters, a predetermined shape and apredetermined size on the real space of the target object, and thesecond position on the real space relating to the first position,present a first image indicating a target object on the captured imageappearing in a camera defined by the camera parameters upon disposingthe target object in a second position in the captured image relating tothe first position.

A third image processing device provided by the present inventionincludes: a first display means configured to display a captured imagecaptured by a camera; a parameter acquisition means configured toacquire a cameral parameter indicates a position and an attitude of thecamera; an input means configured to accept inputting of a dot or a linerelating to the captured image; and a second display means configured todisplay the first image indicating the dot or a line mapped on a planerepresenting a ground surface is viewed from a direction vertical to theplane, based on the camera parameter, a position of the dot or the lineon the captured image.

A first image processing method provided by the present inventionincludes: an input step of accepting inputting of an operation formovement, on a captured image captured by a camera, to a first imagethat is superimposed on the captured image on the basis of predeterminedcamera parameters indicating a position and attitude of the camera andindicates a target object having a predetermined shape and apredetermined size set on a real space; and a presentation step ofpresenting the first image indicating the target object in a manner ofview relating to a position on the captured image after the movement onthe basis of the camera parameters.

A second image processing method provided by the present inventionincludes: a display step of displaying a captured image captured by acamera; a parameter acquisition step of acquiring a cameral parameterindicating a position and an attitude of the camera; an input step ofaccepting designation of a first position in the captured image; and apresentation step of, based on the camera parameters, a predeterminedshape and a predetermined size on the real space of the target object,and the second position on the real space relating to the firstposition, presenting a first image indicating a target object on thecaptured image appearing in a camera defined by the camera parametersupon disposing the target object in a second position in the capturedimage relating to the first position.

A third image processing method provided by the present inventionincludes: a first display step of displaying a captured image capturedby a camera; a parameter acquisition step of acquiring a cameralparameter indicates a position and an attitude of the camera; an inputstep of accepting inputting of a dot or a line relating to the capturedimage; and a second display step of displaying the first imageindicating the dot or a line mapped on a plane representing a groundsurface is viewed from a direction vertical to the plane, based on thecamera parameter, a position of the dot or the line on the capturedimage.

A program provided by the present invention cause a computer to operateas the first image processing device, the second image processingdevice, or the third image processing device.

Advantageous Effects of Invention

According to the present invention, a technique enabling the user toeasily confirm whether camera parameters are appropriate is provided.

BRIEF DESCRIPTION OF DRAWINGS

The above-described object and other objects as well as features andadvantages will become further apparent from the following descriptionof preferred example embodiments and the following accompanyingdrawings.

FIG. 1 is a block diagram illustrating an image processing deviceaccording to a first example embodiment.

FIG. 2 is a diagram illustrating a situation where an image processingdevice has presented a predetermined object on a captured image.

FIG. 3 is a flowchart illustrating a flow of processing executed by theimage processing device of the first example embodiment.

FIG. 4 is a diagram illustrating a captured image in which a first imagehas been presented by a presentation unit.

FIG. 5 is a block diagram illustrating a hardware configuration of animage processing device.

FIG. 6 is a diagram illustrating a situation where a first imageindicating a target object of a planar shape is presented on a capturedimage.

FIG. 7 is a block diagram illustrating an image processing deviceaccording to a second example embodiment.

FIG. 8 is a diagram illustrating a captured image in which a line isinput via a second input unit.

FIG. 9 is a diagram illustrating an image indicating a situation where adotted line mapped on a plane representing a ground surface is viewedfrom a direction vertical to the plane.

FIG. 10 is a diagram illustrating an image in which a position and afield of view of a camera have been presented together with a projectiveline illustrated in FIG. 9(a).

FIG. 11 is a flowchart illustrating a flow of processing executed by theimage processing device of the second example embodiment.

FIG. 12 is a diagram illustrating a situation where error information ispresented on a captured image.

FIG. 13 is a diagram illustrating a situation where the user moves atarget object on a captured image.

FIG. 14 is a diagram illustrating a situation where a target object ismoved across a plurality of areas having different heights.

FIG. 15 is a block diagram illustrating an image processing deviceaccording to a third example embodiment.

FIG. 16 is a flowchart illustrating a flow of processing executed by theimage processing device of the third example embodiment.

FIG. 17 is a diagram illustrating a projective line of a target objectpresented on a captured image on the plane representing the groundsurface illustrated in FIG. 9(a).

DESCRIPTION OF EMBODIMENTS

Hereinafter, example embodiments of the present invention will bedescribed using the accompanying drawings. In all the drawings, the samecomponents are assigned with the same reference signs, and descriptionthereof will be omitted, as appropriate.

First Example Embodiment

FIG. 1 is a block diagram illustrating an image processing device 2000according to a first example embodiment. In FIG. 1 , an arrow indicatesa flow of information. Further, in FIG. 1 , each block does notrepresent a configuration of a hardware unit but represents aconfiguration of a function unit.

The image processing device 2000 includes a display unit 2020, aparameter acquisition unit 2040, an input unit 2060, and a presentationunit 2080.

The display unit 2020 displays a captured image captured by a camera.The parameter acquisition unit 2040 acquires camera parametersindicating a position and attitude or the like of the camera. The cameraparameters may include a parameter other than the position and attitudeof the camera. The parameter other than the position and attitude of thecamera will be described later.

The input unit 2060 accepts a designation of a first position on acaptured image. The presentation unit 2080 generates a first imageindicating a target object on the captured image appearing in a cameradefined by the camera parameters upon disposing the target object in asecond position on a real space relating to the first position. In otherwords, the first image is an image indicating how the target objectlooks when viewed from a point of view of the camera defined by thecamera parameters. Further, it is possible to determine the secondposition on the real space from the camera parameters and heightinformation of the first position and the second position. “Disposing atarget object in a second position” means that it is assumed that thetarget object exists in a position (the second position) on a real spacerelating to the first position on the captured image. The presentationunit 2080 generates the first image using the camera parameters, apredetermined shape and a predetermined size on the real space of thetarget object, and the second position. Further, the presentation unit2080 presents the generated first image in the first position on thecaptured image. The target object is a virtual object having a planarshape or a solid shape. The predetermined size and the predeterminedshape set for the target object are a size and a shape in which a realworld is assumed. The predetermined size and the predetermined shape maybe input by the user or may be previously stored in the inside or theoutside of the image processing device 2000.

Using FIG. 2 , specific description will be made. FIG. 2 is a diagramillustrating a situation where the image processing device 2000 haspresented a predetermined object on a captured image. In FIG. 2 , thepredetermined object is a rectangular parallelepiped 20. FIG. 2(a)illustrates a situation where the rectangular parallelepiped 20 isviewed at an appropriate angle. As illustrated in FIG. 2(a), a size ofthe rectangular parallelepiped 20 is 30 cm in width and depth and 170 cmin height. The rectangular parallelepiped 20 in this example is anobject in which a shape and size of an average person are simplified.

FIG. 2(b) is a diagram in which the image processing device 2000 haspresented the rectangular parallelepiped 20 on a captured image 10. Afirst position 30 indicates a first position input to the input unit2060. The presentation unit 2080 presents a first image 40 in the firstposition 30. The first image 40 is an image indicating in a pseudomanner, when the rectangular parallelepiped 20 disposed in a positionequivalent to the first position 30 in a real world is image-captured bya camera specified by camera parameters, the rectangular parallelepiped20 appearing in the camera.

<Flow of Processing>

FIG. 3 is a flowchart illustrating a flow of processing executed by theimage processing device 2000 of the first example embodiment. In stepS102, the display unit 2020 displays a captured image captured by acamera. In step S104, the input unit 2060 accepts a designation of afirst position on the captured image. In step S106, the parameteracquisition unit 2040 acquires camera parameters indicating a positionand attitude or the like of the camera. In step S108, the presentationunit 2080 generates a first image. As described above, the first imageindicates a target object on a captured image upon appearing in a cameraspecified by the camera parameters when being disposed in a secondposition. In step S110, the presentation unit 2080 presents thegenerated first image in the first position on the captured image.

The flow of processing illustrated in FIG. 3 is one example, and a flowof processing executed by the image processing device 2000 is notlimited to the flow illustrated in FIG. 3 . For example, processing(step S106) of acquiring camera parameters may be executed beforeprocessing (step S104) of accepting inputting of a first position.

<Operations and Advantageous Effects>

According to the present example embodiment, the user of the imageprocessing device 2000 views an object presented by the presentationunit 2080, and thereby the user can easily confirm whether cameraparameters appropriately indicate a position and attitude or the like ofa camera (hereinafter, a real camera) having captured a captured imagedisplayed by the display unit 2020. Hereinafter, using FIG. 4 , detaileddescription will be made.

FIG. 4 is a diagram illustrating a captured image in which a first imagehas been presented by the presentation unit 2080. FIG. 4(a) is a diagramin which camera parameters acquired by the parameter acquisition unit2040 indicate a position and attitude approximate to a position andattitude of a real camera. On the other hand, FIG. 4(b) is a diagram inwhich camera parameters acquired by the parameter acquisition unit 2040indicate a position and attitude different from a position and attitudeof a real camera. A target object in FIG. 4 is a rectangularparallelepiped having a height of 170 cm and depth and width of 30 cm inthe same manner as in the case of FIG. 2 .

The first image presented by the presentation unit 2080 is presented ona captured image as if a target object disposed in a place appearing ona captured image has been image-captured by a camera installed in aposition and attitude indicated by camera parameters. Therefore, whenthe camera parameters indicate a position and attitude approximate to aposition and attitude of a real camera, there is no feeling ofstrangeness in a manner of view or the like depending on a size andangle when a person, an object, or the like appearing on the capturedimage and the first image are compared. A height of the target objectis, for example, 170 cm, and therefore when the target object and aperson are compared, it is conceivable that heights to substantially thesame extent are obtained.

In FIG. 4 , a transverse side of a person appearing on the capturedimage 10 is designated as a first position, and therefore the firstimage 40 is presented in a transverse side of the person. In FIG. 4(a),in any position, sizes of a person and a rectangular parallelepipedindicated by the first image 40 are substantially the same, resulting inno feeling of strangeness. Further, in FIG. 4(a), in the same manner asin a case where a person, a wall, and the like appear by being lookeddown from a front-diagonally upward side, a rectangular parallelepipedindicated by each first image 40 is also looked down from afront-diagonally upward side, and therefore there is no feeling ofstrangeness also in a manner of view depending on an angle of eachrectangular parallelepiped.

In contrast, in FIG. 4(b), there is a feeling of strangeness in a mannerof view caused by a size and angle of a rectangular parallelepipedindicated by the first image 40. For example, a height of a rectangularparallelepiped indicated by a first image 40-10 is approximately twice aheight of a person, and therefore it is difficult to say that the firstimage 40-10 indicates an object (the rectangular parallelepiped 20)having a height of 170 cm disposed in a place appearing on a capturedimage 10-2. Further, differently from a rectangular parallelepipedindicated by each first image 40 presented by a captured image 10-1, inthe captured image 10-2, a top surface of every rectangularparallelepiped is visible and appears in a manner of view so as to belooked down from a close proximity. In this manner, from a size andangle of each rectangular parallelepiped indicated by the first image40, it is predictable that a depression angle of a sight line directionof a camera indicated by camera parameters in FIG. 4(b) has come to belarger than a depression angle of a sight line direction of a realcamera.

As illustrated in FIG. 4 , the user may designate a plurality of firstpositions and dispose a plurality of target objects within one capturedimage.

As described above, according to the image processing device 2000 of thepresent example embodiment, the user using the image processing device2000 compares a first image presented by the presentation unit 2080 anda captured image and thereby can easily grasp whether camera parametersacquired by the parameter acquisition unit 2040 indicate a position andattitude approximate to a position and attitude of a camera havingcaptured the captured image. When it is possible to confirm that aposition and attitude approximate to a position and attitude of a camerahaving captured a captured image are indicated, the user can determinethat a combination between the camera parameters and a video of amonitoring camera is usable. Conversely, when it is possible to confirmthat a position and attitude approximate to a position and attitude of acamera having captured a captured image are not indicated,countermeasures such that camera parameters are estimated again and aposition and attitude of a real camera are corrected may be taken.

Hereinafter, the image processing device 2000 of the present exampleembodiment will be described in more detail.

<Hardware Configuration Example>

Each function configuration unit of the image processing device 2000 maybe realized by a hardware component (e.g. a hard-wired electroniccircuit) that realizes each function configuration unit or may berealized by a combination between a hardware component and a softwarecomponent (e.g. a combination between an electronic circuit and aprogram that controls the circuit).

FIG. 5 is a block diagram illustrating a hardware configuration of theimage processing device 2000. The image processing device 2000 includesa bus 1020, a processor 1040, a memory 1060, a storage 1080, and aninput/output interface 1100. The bus 1020 is a data transmission channelin order for the processor 1040, the memory 1060, the storage 1080, andthe input/output interface 1100 to mutually execute datatransmission/reception. However, a method for mutually connecting theprocessor 1040 and the like is not limited to bus connection. Theprocessor 1040 is an arithmetic processing unit such as a CPU (CentralProcessing Unit), a GPU (Graphics Processing Unit), or the like, forexample. The memory 1060 is a memory such as a RAM (Random AccessMemory), a ROM (Read Only Memory), or the like, for example. The storage1080 is a storage device such as a hard disk, an SSD (Solid StateDrive), a memory card, or the like, for example. Further, the storage1080 may be a memory such as a RAM, a ROM, or the like. The input/outputinterface 1100 is an input/output interface in order for the imageprocessing device 2000 to transmit/receive data between itself and aninput device, an external device, or the like. The image processingdevice 2000 acquires, for example, the captured image and the firstposition via the input/output interface 1100. Further, the imageprocessing device 2000 outputs, for example, a captured image presentinga first image via the input/output interface.

The storage 1080 stores a program for realizing a function of the imageprocessing device 2000. Specifically, the storage stores program modulesfor realizing functions of the display unit 2020, the parameteracquisition unit 2040, the input unit 2060, and the presentation unit2080, respectively. The processor 1040 executes these program modulesand thereby realizes the functions of the display unit 2020, theparameter acquisition unit 2040, the input unit 2060, and thepresentation unit 2080, respectively. When executing the modules, theprocessor 1040 may read the modules onto the memory 1060 and execute themodules or may execute the modules without being read onto the memory1060.

The hardware configuration of the image processing device 2000 is notlimited to the configuration illustrated in FIG. 5 . For example, eachprogram module may be stored on the memory 1060. In this case, the imageprocessing device 2000 may not include the storage 1080.

<Details of Camera Parameters>

As described above, camera parameters may include a parameter other thana position and attitude of a camera. The camera parameters include, forexample, an internal parameter indicating an internal characteristic ofa camera such as a focal length, lens distortion, coordinates of acenter of an image, and the like. The position and attitude of a camerais an external parameter indicating an external characteristic of thecamera. The camera parameters may be calculated by associatingtwo-dimensional coordinates on a captured image with three-dimensionalcoordinates on a real space.

When camera parameters are used, mutual transformation between thetwo-dimensional coordinates on the captured image and thethree-dimensional coordinates on a real space may be made. However, itdifficult that the two-dimensional coordinates on the captured imageuniquely determines, by itself, the three-dimensional coordinates on thereal space relating to the two-dimensional coordinates. To uniquelydetermine the three-dimensional coordinates on the real space relatingto the two-dimensional coordinates on the captured image, it isnecessary to specify, for example, any one of an x-coordinate, ay-coordinate, and a z-coordinate of the three-dimensional coordinates.The image processing device 2000 of the present example embodimentspecifies height information (the z-coordinate) of the second positionon the real space and thereby uniquely determines the second position onthe real space relating to the first position on the captured image. Inthe present example embodiment, an origin on the real space is set on aground surface immediately below a camera, the x-coordinate and they-coordinate are set in a width direction and a depth direction parallelto the ground surface, respectively, and the z-coordinate is set in adirection vertical to the ground surface to make description. Atechnique for executing mutual transformation between coordinates on animage and coordinates on a real space using camera parameters is a knowntechnique and is described in, for example, NPL 1. Therefore, furtherdetailed description on this technique will be omitted.

There are various methods in which the parameter acquisition unit 2040acquires camera parameters. The parameter acquisition unit 2040receives, for example, camera parameters transmitted from an externaldevice. Further, the parameter acquisition unit 2040 accepts, forexample, manual inputting of camera parameters. Further, the parameteracquisition unit 2040 reads, for example, camera parameters from astorage device storing camera parameters.

<Details of Display Unit 2020>

The display unit 2020 displays a captured image on a display screen suchas a display and the like. The display screen may be a stationarydisplay or may be a portable display included in a mobile terminal andthe like.

<Details of Input Method of First Position>

The input unit 2060 may accept a designation of a first position usingvarious methods capable of specifying a position on a captured image.The input unit 2060 accepts, for example, an operation (a clickoperation or the like) for designating any position on a captured imageby an input device such as a mouse and the like. Further, when acaptured image is displayed on a touch panel, the input unit 2060accepts touch inputting or the like for any position on the capturedimage. Further, the input unit 2060 may accept inputting of coordinatesindicating a position on a captured image.

<Details of Target Object>

A target object is an object having, for example, a predetermined sizeand shape on a real space. Information defining a predetermined targetobject that is, for example, “a rectangular parallelepiped having aheight of 170 cm and depth and width of 30 cm” as described above ispreviously stored in the inside or outside of the image processingdevice 2000. In this case, the presentation unit 2080 uses thispredetermined handling object.

Further, the image processing device 2000 may include a function foraccepting inputting of information defining a target object. In thiscase, the device may accept information indicating both a shape and asize on a real space of the target object or may accept informationindicating only any one of the shape and the size. In the latter case,the shape of the target object is previously determined as a shape of arectangular parallelepiped, for example, and a designation of the size(depth and width and a height) is accepted from the user.

The shape of the target object is not limited to a rectangularparallelepiped. The target object may be, for example, conical orspherical. Further, the target object may be an object indicating ashape of a person, an animal, or the like such as an avatar and thelike.

Further, the target object may have a planar shape. FIG. 6 is a diagramillustrating a situation where a first image 40 indicating a targetobject of a planar shape is presented on a captured image 10. In thiscase, the user designates, for example, depth and width of a plane. Whencamera parameters appropriately indicate a position and attitude or thelike of a real camera, a first image presented by the presentation unit2080 becomes parallel to a ground surface. The user compares the firstimage presented by the presentation unit 2080 with the ground surfaceappearing on the captured image 10 and checks whether a planerepresented by the first image 40 is parallel to the ground surface, andthereby may easily confirm whether the camera parameters appropriatelyindicate the position and attitude or the like of the real camera.Further, the size of the plane is designated, and therefore, when anobject or the like of a known size appearing within a captured image iscompared with an appearance and a size on an image and a feeling ofstrangeness is confirmed, it is possible to easily confirm whether thecamera parameters appropriately indicate a position and attitude or thelike of the real camera.

<Details of Presentation Unit 2080>

As described above, the presentation unit 2080 generates, when a targetobject disposed in a second position appears in a camera determined bythe camera parameters, an image indicating the target object on acaptured image. The presentation unit 2080 executes, for example, thefollowing processing.

First, the presentation unit 2080 calculates a second position on a realspace relating to a first position on a target image. As describedabove, it is difficult that a first position (two-dimensionalcoordinates) on a target image uniquely determines, by itself, a secondposition (three-dimensional coordinates) on a real space relating to thefirst position. Therefore, the presentation unit 2080 acquiresinformation (a z-coordinate of the second position) indicating a heightof the second position. The height information of the second positionindicates, for example, a height (z=0) of a ground surface on the realspace. When the height information of the second position is specifiedin this manner, a position on the real space relating to the firstposition on the target image is uniquely determined. The presentationunit 2080 calculates three-dimensional coordinates of the secondposition using two-dimensional coordinates of the first position, theheight information of the second position, and camera parameters. Asdescribed above, when these pieces of information are used,two-dimensional coordinates on a captured image can be transformed tothree-dimensional coordinates on a real space. The height information ofthe second position can be previously provided for the presentation unit2080 or can be supplied from the outside. Alternatively, the heightinformation of the second position may be set as a different height foreach of a plurality of areas within a target image.

The presentation unit 2080 generates a first image indicating a targetobject to be presented on the captured image. When the target objecthas, for example, a shape of a rectangular parallelepiped or a cone, thepresentation unit 2080 calculates coordinates of each apex of the targetobject to be presented on the captured image to generate the firstimage. Specifically, the presentation unit 2080 transformsthree-dimensional coordinates of each apex in which the target object isdisposed in the second position on the real space to two-dimensionalcoordinates of each apex on the captured image, using the cameraparameters. The presentation unit 2080 generates the first image byconnecting each apex with a straight line or the like.

An angle of the target object disposed in the real space is optional.The presentation unit 2080 assumes that the target object has beendisposed in the second position such that, for example, in an xyz spacerepresenting the real space, a width-direction side of the target objectis parallel to the x-axis, a depth-direction side thereof is parallel tothe y-axis, and a height-direction side thereof is parallel to thez-axis. Directions of these sides may be previously determined, ordesignations therefor by the user may be accepted. When, for example, inthe captured image 10 of FIG. 6 , a target object of a planar shape isused, a depth-direction side is matched with a line on a ground surface,and thereby it becomes possible to easily determine whether the targetobject and the ground surface are parallel to each other. In additionthereto, when, for example, a lattice of a predetermined width on a realspace is drawn in a target object, a depth-direction side of the targetobject is matched with a line or the like of a tile having a known sizeof a floor face, and thereby a size of the tile may be measured. Thesize is confirmed, and thereby determination is more easily performed.Therefore, the image processing device 2000, for example, enables theuser to rotate a target object being presented on the captured image 10using a mouse or the like. When, for example, the target object beingpresented on the captured image 10 or a periphery thereof has beendragged by a mouse or the like, the image processing device 2000determines a direction of rotating the target object in accordance witha direction of the drag. For example, a rotation direction upon beingdragged in a left direction is regarded as clockwise rotation, and arotation direction upon being dragged in a right direction is regardedas counter-clockwise rotation. Further, the image processing device 2000determines an angle of rotation of the target object in accordance witha distance of the drag. In this case, a relation between a distance of adrag and an angle of rotation is previously defined. The imageprocessing device 2000 rotates the target object on the basis of thedetermined direction and angle around a straight line (e.g. a straightline parallel to the z-axis), as a rotation axis, passing through thesecond position. The user disposes the depth-direction side of thetarget object along a line of the ground surface and compares the targetobject on the captured image 10 with the ground surface. The secondposition is not limited to an internal point of the target object andmay be located externally.

Further, the presentation unit 2080 may accept an operation for moving atarget object on the captured image 10. The user moves the target objecton the captured image 10, for example, by an operation such as “draggingon the captured image 10 by the right button of a mouse.” In this case,the input unit 2060 repeatedly acquires a position of a moving mousepointer as the above-described first position. This acquisition isexecuted, for example, at a predetermined time interval. Thepresentation unit 2080 presents, in a first position on the capturedimage 10 newly acquired by the input unit 2060, the first image 40 newlygenerated on the basis of the first position, a fixedly obtained cameraparameters, and height information of a second position. Further, thepresentation unit 2080 deletes, from the captured image 10, the firstimage 40 having been presented in a first position acquired before thefirst position. By doing so, from a point of view of the user, thetarget object appears to be moving on a space appearing on the capturedimage 10.

FIG. 13 is a diagram illustrating a situation where the user moves atarget object on the captured image 10. In FIG. 13 , a trajectory 170indicates a trajectory in which the user has moved a target object. Afirst position 30-1 to a first position 30-5 indicate positions on thetrajectory 170, respectively. A first image 40-1 to a first image 40-5indicate first images 40 presented in the first position 30-1 to thefirst position 30-5, respectively. The first image 40 drawn with dottedlines indicates the first image 40 having already disappeared from thecaptured image 10, and the first image 40 drawn with solid linesindicates the first image 40 being currently presented. In FIG. 13 ,since a currently designated first position 30 is the first position30-5, the first image 40-5 is being presented and the first image 40-1to the first image 40-4 have disappeared.

As illustrated in FIG. 13 , for example, the user moves a target objectso as to pass through a transverse side of a person or the likeappearing on the captured image 10 and thereby confirms whether there isno feeling of strangeness in a manner of view of the target object. Inthe case of FIG. 13 , when there is no feeling of strangeness in a sizeand direction of the target object even upon moving the target object toa transverse side of any person, it is conceivable that cameraparameters acquired by the parameter acquisition unit 2040 indicate aposition and attitude approximate to a position and attitude of a camerahaving captured the captured image 10. When such a moving operation isprovided, the user can easily verify, for various positions on thecaptured image 10, whether there is no feeling of strangeness in amanner of view of the target object. Specifically, when a manner of viewof the target object is provided via continuous movement, rightfulnessand a feeling of strangeness based on human visual sense is furtheremphasized, resulting in an effective function for verification.

Further, as illustrated in FIG. 14 , for example, in a captured image inwhich areas having a step as in stairs appear, height information may beset for each area having a step. In this case, as a trajectory isillustrated in FIG. 14 , by moving a target object on an image, the usermay easily verify whether there is no feeling of strangeness in a mannerof view of the target object seamlessly including the steps.

Second Example Embodiment

FIG. 7 is a block diagram illustrating an image processing device 2000according to a second example embodiment. In FIG. 7 , an arrow indicatesa flow of information. Further, in FIG. 7 , each block does notrepresent a configuration of a hardware unit but represents aconfiguration of a function unit.

The image processing device 2000 of the second example embodimentincludes a display unit 2020, a parameter acquisition unit 2040, asecond input unit 2100, and a second display unit 2120. Functionsincluded in the display unit 2020 and the parameter acquisition unit2040 of the present example embodiment are the same as the functionsincluded in the display unit 2020 and the parameter acquisition unit2040 described in the first example embodiment, respectively.

The second input unit 2100 accepts inputting of a point or line to acaptured image displayed by the display unit 2020. The second displayunit 2120 displays, on the basis of camera parameters, a position on thecaptured image of the input point or line, and height information on areal space of the input point or line, an image indicating the point orline upon mapping on a plane parallel to a ground surface. In otherwords, the second display unit 2120 displays, when it is assumed thatthe input point or line within the captured image exists within a fieldof view of a camera having captured the captured image, an image inwhich the point or line assumed to exist within the field of view of thecamera is mapped on the plane parallel to the ground surface. The seconddisplay unit 2120 may perform display for the same display as a displayor the like on which a captured image is being displayed by the displayunit 2020 or may perform display for a different display or the like.

The height information of the input point or line on the real space maybe previously provided for the second display unit 2120 or may be inputto the second input unit 2100 together with the point or line. When theheight information on the real space of the input point or line ispreviously provided for the second display unit 2120, the heightinformation is set as, for example, a height (e.g. height information(z-coordinate)=0) of a ground surface on the real space.

As described above, the second display unit 2120 maps a point or lineexisting on a captured image on a plane parallel to a ground surface ina real space. First, a mapping method of a point is described below. Thesecond display unit 2120 transforms two-dimensional coordinates of apoint on a captured image to three-dimensional coordinates on a realspace. As described above, three-dimensional coordinates on the realspace relating to two-dimensional coordinates on the captured image arenot uniquely determined. Therefore, the second display unit 2120 usesheight information of the input point. Specifically, it is assumed thatthe height information on the real space of the input point is givenheight information. Thereby, the second display unit 2120 may uniquelytransform two-dimensional coordinates on the captured image tothree-dimensional coordinates on the real space. A position of the inputpoint on the plane parallel to the ground surface on the real space isrepresented by a width-direction coordinate and a depth-directioncoordinate (the x-coordinate and the y-coordinate except thez-coordinate indicating height) of calculated three-dimensionalcoordinates.

As described in the first example embodiment, a technique forcalculating, on the basis of camera parameters, two-dimensionalcoordinates of a point on a captured image, and height information on areal space of the point, three-dimensional coordinates on the real spacerelating to the two-dimensional coordinates is a known technique.Therefore, detailed description on this technique will be omitted.

A principle of processing of mapping a line input onto a captured imageon a plane parallel to a ground surface in a real space is the same asthe above-described principle of processing of mapping a point. Thesecond display unit 2120 maps, for example, each of two or more points(e.g. points of both ends) existing on an input line on a plane parallelto a ground surface in a real space. The second display unit 2120connects these mapped points with a line such as a straight line and thelike. By doing so, the line input onto the captured image is mapped onthe plane parallel to the ground surface in the real space.

Hereinafter, a utilization method of the image processing device 2000 ofthe second example embodiment will be described.

The user of the image processing device 2000 inputs, for example, apattern in a real world and a line tracing a border between a wall and aground surface to the second input unit 2100. FIG. 8 is a diagramillustrating a captured image 10 in which a line has been input via thesecond input unit 2100. A dotted line 90 represents a line input to thesecond input unit 2100. A pattern 100 is a line drawn on a groundsurface on a real world appearing on a captured image. A pattern 100-1and a pattern 100-2 are lines parallel to each other on the real world.A border 110 is a border between a wall and the ground surface on thereal world appearing on the captured image. A border 110-1 and a border110-2 vertically intersect with each other on the real world.

The second display unit 2120 maps the dotted line 90 on a plane parallelto the ground surface. The second display unit 2120 displays a situationwhere the dotted line 90 mapped on the plane parallel to the groundsurface is viewed from a direction vertical to the plane. FIG. 9 is adiagram illustrating an image representing a situation where the dottedline 90 mapped on a plane representing a ground surface is viewed from adirection vertical to the plane. FIG. 9(a) is a diagram in which cameraparameters indicate a position and attitude approximate to a positionand attitude of a real camera. As described above, in a real world (in aplace appearing on a captured image), the pattern 100-1 and the pattern100-2 are lines drawn parallel to each other. Therefore, in FIG. 9(a) inwhich camera parameters indicate a position and attitude approximate toa position and attitude of a real camera, a projective line 120-1 inwhich a dotted line 90-1 is mapped on a plane representing a groundsurface and a projective line 120-2 in which a dotted line 90-2 ismapped on the plane representing the ground surface are parallel orsubstantially parallel to each other. Further, as described above, in areal world (in a place appearing on a captured image), a border 110-3and a border 110-4 vertically intersect with each other. Therefore, inFIG. 9(a), a projective line 120-3 in which a dotted line 90-3 is mappedon the plane representing the ground surface and a projective line 120-4in which a dotted line 90-4 is mapped on the plane representing theground surface intersect with each other vertically or at asubstantially vertical angle.

On the other hand, FIG. 9(b) is a diagram in which camera parametersindicate a position and attitude different from a position and attitudeof a real camera. In this case, the projective line 120-1 and theprojective line 120-2 may not have a parallel or substantially parallelrelation, or the projective line 120-3 and the projective line 120-4 maynot have a vertical or substantially vertical relation.

In this manner, when the user using the captured image illustrated inFIG. 8 uses the pattern 100 and the border 110 in which a relation in areal world is known or easily predicted and views a result in whichthese are displayed by the second display unit 2120, the user may easilyconfirm whether camera parameters appropriately indicate a position andattitude of a real camera.

The method for using a pattern and the like on a ground surface is notlimited to the above-described method. A method for inputting aplurality of points onto the pattern 100-1 and confirming whether theplurality of points are disposed on a straight line is conceivable, forexample.

Further, the image processing device 2000 of the present exampleembodiment may map and present, on the plane, a target object beingpresented on a captured image in the first example embodiment. FIG. 17is a diagram illustrating a projective line 180 of a target objectpresented on a captured image on the plane representing the groundsurface illustrated in FIG. 9(a). FIG. 17(a) is a case in which theprojective line 180 of the target object is presented when a first imageindicating a still target object is presented on a captured image (e.g.FIG. 2(b)). On the other hand, FIG. 17(b) is a case in which theprojective line 180 of the target object is moved in accordance withmovement of the target object on a captured image when an operation formoving the target object is being executed (e.g. FIG. 13 ). A trajectory190 represents a trajectory of movement of the projective line 180.

Further, when an object (a manhole or the like) in which an originalshape is understandable appears on a ground surface of a captured image,a line tracing the shape may be input to the second input unit 2100.When camera parameters indicate a position and attitude approximate to aposition and attitude of a real camera, a shape of a line displayed bythe second display unit 2120 represents a shape close to an originalshape of a traced object. When, for example, a line is input so as totrace a manhole appearing on a captured image, a shape of the linedisplayed by the second display unit 2120 becomes a perfect circle or ashape close to a perfect circle. On the other hand, when cameraparameters indicate a position and attitude different from a positionand attitude of a real camera, a shape of a line presented by the seconddisplay unit 2120 becomes a shape (e.g. an elliptical shape) differentfrom a perfect circle.

Further, the second display unit 2120 may present a position and a fieldof view of a camera on an image, together with a point and a line mappedon a plane parallel to a ground surface. FIG. 10 is a diagramillustrating an image in which a position and a field of view of acamera are presented, together with the projective lines illustrated inFIG. 9(a). In FIG. 10 , a camera position 150 represents a position ofthe camera, and a field of view 160 represents a field of view of thecamera.

A system setter or the like handling the image processing device 2000 ofthe second example embodiment views a position relation of a point and aline mapped on a plane parallel to a ground surface and thereby confirmswhether camera parameters appropriately indicate a position and attitudeor the like of a real camera. As illustrated in FIG. 10 , when aposition and a field of view of a camera are presented together with apoint and a line mapped on a plane parallel to a ground surface, thesystem setter or the like may further grasp a positon relation betweenthe mapped point and line and the position and the field of view of thecamera. Therefore, the system setter or the like may more easily andaccurately confirm whether the camera parameters appropriately indicatethe position and attitude or the like of the real camera.

<Flow of Processing>

FIG. 11 is a flowchart illustrating a flow of processing executed by theimage processing device 2000 of the second example embodiment.Processing executed in steps S102 and S106 is the same as the processingexecuted in steps S102 and S106 of FIG. 3 . In step S202, the secondinput unit 2100 accepts inputting of a point or line to a captured imagedisplayed by the display unit 2020. In step S204, the second displayunit 2120 displays an image indicating the point or line upon mapping ona plane parallel to a ground surface.

<Operations and Advantageous Effects>

According to the image processing device 2000 of the present exampleembodiment, the user inputs a line or the like that easily specifies anoriginal shape or a position relation to a captured image and checkswhether a line or the like displayed by the second display unit 2120satisfies the original shape or the position relation, and thereby mayeasily confirm whether camera parameters appropriately indicate aposition and attitude or the like of a real camera.

Third Example Embodiment

FIG. 15 is a block diagram illustrating an image processing device 3000according to a third example embodiment. In FIG. 15 , an arrow indicatesa flow of information. Further, in FIG. 15 , each block does notrepresent a configuration of a hardware unit but represents aconfiguration of a function unit.

The image processing device 3000 of the third example embodimentincludes an input unit 3020 and a presentation unit 3040. The input unit3020 accepts inputting of an operation for moving a first image beingpresented on a captured image captured by a camera. The first image isan image in which a target object having a predetermined shape and apredetermined size on a real space is superimposed on the captured imageon the basis of predetermined camera parameters indicating a positionand attitude of the camera. When, for example, a position on thecaptured image in which the first image is being presented is designatedas a position A, the first image is equivalent to a first imagepresented by the presentation unit 2080 upon designating the position Aas a first position in the image processing device 2000 of the firstexample embodiment. A target object in the third example embodiment isthe same as the target object described in the first example embodiment.Further, predetermined camera parameters in the third example embodimentis the same as the camera parameters described in the first exampleembodiment.

The presentation unit 3040 presents, on the basis of the cameraparameters, a first image indicating a target object in a manner of viewrelating to a position on the captured image after the movement. Amethod in which the presentation unit 3040 presents a first imagerelating to a target object to be moved is the same as “the method inwhich the presentation unit 2080 presents the first image 40 relating toa target object to be moved on the captured image 10” described in thefirst example embodiment.

A hardware configuration of the image processing device 3000 is the sameas the hardware configuration of the image processing device 2000.

<Flow of Processing>

FIG. 16 is a flowchart illustrating a flow of processing executed by theimage processing device 3000 of the third example embodiment. In stepS302, the input unit 3020 accepts inputting of an operation for movementto a first image superimposed on a captured image. In step S304, thepresentation unit 3040 presents, on the basis of camera parameters, thefirst image indicating the target object in a manner of view relating toa position on the captured image after the movement.

The flow of processing illustrated in FIG. 16 is one example, and a flowof processing executed by the image processing device 3000 is notlimited to the flow illustrated in FIG. 16 .

<Operations and Advantageous Effects>

According to the present example embodiment, as illustrated, forexample, in FIG. 13 or FIG. 14 , the user moves a target object so as topass through a transverse side of a person or the like appearing on thecaptured image 10 and thereby may easily confirm whether there is nofeeling of strangeness in a manner of view of the target object.Specifically, when a manner of view of a target object is provided viacontinuous movement, rightfulness and a feeling of strangeness based onhuman visual sense are further emphasized, resulting in an effectivefunction for verification.

Modified Examples

The image processing device 2000 may include functions as describedbelow. The image processing device 2000 including the followingfunctions is expressed as an image processing device 2000 of a firstmodified example. The image processing device 2000 of the first modifiedexample may include the functions of the image processing device 2000 ofthe above-described first and second example embodiments or may notinclude these functions.

As describe above, for estimation of camera parameters, used is a methodin which “a calibration pattern or an object equivalent thereto isimage-captured by a camera, and estimation is performed on the basis ofan association relation between three-dimensional coordinates of thecalibration pattern in a real world and two-dimensional coordinates ofthe calibration pattern of the captured image” (NPL 1). Specifically,camera parameters are calculated so as to reduce, using estimated cameraparameters, an error (re-projection error) between two-dimensionalcoordinates upon projecting three-dimensional coordinates of acalibration pattern in a real world on a captured image andtwo-dimensional coordinates of the calibration pattern appearing on thecaptured image. There is, for example, a method for calculatingestimation values of camera parameters so as to minimize a square sum oferrors.

Commonly, when a system setter or the like handling the image processingdevice 2000 performs work for estimating camera parameters using theabove-described calibration, the system setter or the like views onlycamera parameters as an estimation result and does not view the errorthat is an interim progress. However, when the error that is an interimprogress is caused to be viewed by the system setter or the like, it isconceivable that accuracy in estimation of camera parameters may beenhanced. When, for example, positions having large errors areconcentrated on an edge of a captured image, it is conceivable that anerror is increased due to a cause resulting from an input error of acorresponding point or lens distortion. In such a case, when a selectionmanner of a calibration pattern is changed so as not to use acalibration pattern image-captured in a position within a predetermineddistance from an edge of an image to estimate camera parameters,accuracy of the camera parameters may be enhanced.

The image processing device 2000 presents, for each position where acalibration pattern is image-captured, the error with respect to thecalibration pattern image-captured in the position in a periphery of aposition on a captured image relating to the position. FIG. 12 is adiagram illustrating a situation where information (error information140) indicating an error is presented on a captured image. In FIG. 12 ,to obtain a calibration pattern, a person is used. Specifically, a line130 connecting the feet and the head of a person substantially standingerect is used as a calibration pattern. The error information 140presented in a transverse side of the line 130 indicates a re-projectionerror relating to the line 130.

The image processing device 2000 may map the calibration pattern on aground surface on the basis of the technique described in the secondexample embodiment and display the error in association with thecalibration pattern mapped on the ground surface.

While the example embodiments of the present invention have beendescribed with reference to the drawings, these example embodiments areillustrative of the present invention, and various constitutions otherthan the above are employable.

Hereinafter, examples of reference modes will be supplementarily noted.

1. An image processing device includes:

-   -   an input means configured to accept inputting of an operation        for movement, on a captured image captured by a camera, to a        first image that is superimposed on the captured image on the        basis of predetermined camera parameters indicating a position        and attitude of the camera and indicates a target object having        a predetermined shape and a predetermined size set on a real        space; and    -   a presentation means configured to present the first image        indicating the target object in a manner of view relating to a        position on the captured image after the movement on the basis        of the camera parameters.        2. The image processing device according to 1., wherein    -   the input means accepts an operation for the movement by        repeatedly accepting a designation of a first position on the        captured image, and    -   the presentation means generates, when a certain first position        is designated, on the basis of the camera parameters, a        predetermined shape and a predetermined size on a real space of        the target object, and a second position on the real space        relating to the first position, a first image indicating the        target object on the captured image appearing in a camera        determined by the camera parameters when the target object is        disposed in the second position and presents the generated first        image in the first position on the captured image.        3. The image processing device according to 2., wherein    -   the presentation means        -   acquires height information of the second position and        -   calculates the second position on the basis of the camera            parameters, the first position, and the height information            of the second position.            4. The image processing device according to 3., wherein the            presentation means acquires information indicating a height            of a ground surface in a real space as the height            information of the second position.            5. The image processing device according to 3., wherein the            presentation means acquires pieces of information of            different heights for a plurality of areas on the captured            image, respectively, as the height information of the second            position.            6. The image processing device according to any one of 1. to            5., wherein the target object has a planar shape.            7. The image processing device according to any one of 1. to            6., comprising:    -   a second input means configured to accept inputting of a point        or line to the captured image; and    -   a second display means configured to display a second image        indicating the point or line upon mapping on a plane parallel to        a ground surface, on the basis of the camera parameters, a        position on the captured image of the point or line, and height        information on a real space of the point or line.        8. An image processing device includes:    -   an input means configured to accept a designation of a first        position on a captured image; and    -   a presentation means configured to present, on the basis of        predetermined camera parameters indicating a position and        attitude of a camera, a predetermined shape and a predetermined        size on a real space of a target object, and a second position        on the real space relating to the first position, a first image        indicating the target object on the captured image appearing in        a camera determined by the camera parameters when the target        object is disposed in the second position in the first position        on the captured image.        9. The image processing device according to 8., wherein    -   the input means accepts designations of a plurality of first        positions, and    -   the presentation means presents first images indicating a        plurality of target objects relating to the plurality of first        positions in respective corresponding first positions on the        captured image.        10. The image processing device according to 8. or 9., wherein    -   the input means repeatedly accepts a designation of the first        position, and    -   the presentation means generates, when a certain first positon        is designated, a first image indicating the target object        disposed in a second position on a real space relating to the        first position and presents the generated first image in the        first position on the captured image.        11. An image processing device comprising:    -   an input means configured to accept inputting of a point or line        to a captured image captured by a camera; and    -   a display means configured to display a first image indicating        the point or line upon mapping on a plane parallel to a ground        surface, on the basis of predetermined camera parameters        indicating a position and attitude of the camera, a position on        the captured image of the point or line, and height information        on a real space of the point or line.        12. An image processing method executed by a computer, the        method comprising:    -   an input step of accepting inputting of an operation for        movement, on a captured image captured by a camera, to a first        image that is superimposed on the captured image on the basis of        predetermined camera parameters indicating a position and        attitude of the camera and indicates a target object having a        predetermined shape and a predetermined size set on a real        space; and    -   a presentation step of presenting the first image indicating the        target object in a manner of view relating to a position on the        captured image after the movement on the basis of the camera        parameters.        13. The image processing method according to 12., wherein    -   the input step accepts an operation for the movement by        repeatedly accepting a designation of a first position on the        captured image, and    -   the presentation step generates, when a certain first position        is designated, on the basis of the camera parameters, a        predetermined shape and a predetermined size on a real space of        the target object, and a second position on the real space        relating to the first position, a first image indicating the        target object on the captured image appearing in a camera        determined by the camera parameters when the target object is        disposed in the second position and presents the generated first        image in the first position on the captured image. ps 14. The        image processing method according to 13., wherein    -   the presentation step    -   acquires height information of the second position and    -   calculates the second position on the basis of the camera        parameters, the first position, and the height information of        the second position.        15. The image processing method according to 14., wherein the        presentation step acquires information indicating a height of a        ground surface in a real space as the height information of the        second position.        16. The image processing method according to 14., wherein the        presentation step acquires pieces of information of different        heights for a plurality of areas on the captured image,        respectively, as the height information of the second position.        17. The image processing method according to any one of 12. to        16., wherein the target object has a planar shape.        18. The image processing method according to any one of 12. to        17., including:    -   a second input step of accepting inputting of a point or line to        the captured image, and    -   a second display step of displaying a second image indicating        the point or line upon mapping on a plane parallel to a ground        surface, on the basis of the camera parameters, a position on        the captured image of the point or line, and height information        on a real space of the point or line.        19. An image processing method executed by a computer, the        method comprising:    -   an input step of accepting a designation of a first position on        a captured image; and    -   a presentation step of presenting, on the basis of predetermined        camera parameters indicating a position and attitude of a        camera, a predetermined shape and a predetermined size on a real        space of a target object, and a second position on the real        space relating to the first position, a first image indicating        the target object on the captured image appearing in the camera        determined by the camera parameters when the target object is        disposed in the second position in the first position on the        captured image.        20. The image processing method according to 19., wherein    -   the input step accepts designations of a plurality of first        positions, and    -   the presentation step presents first images indicating a        plurality of target objects relating to the plurality of first        positions in respective corresponding first positions on the        captured image.        21. The image processing method according to 19. or 20., wherein    -   the input step repeatedly accepts a designation of the first        position, and    -   the presentation step generates, when a certain first positon is        designated, a first image indicating the target object disposed        in a second position on a real space relating to the first        position and presents the generated first image in the first        position on the captured image.        22. An image processing method executed by a computer, the        method comprising:    -   an input step of accepting inputting of a point or line to a        captured image captured by a camera; and    -   a display step of displaying a first image indicating the point        or line upon mapping on a plane parallel to a ground surface, on        the basis of predetermined camera parameters indicating a        position and attitude of the camera, a position on the captured        image of the point or line, and height information on a real        space of the point or line.        23. A program that causes a computer to operate as the image        processing device according to any one of 1. to 11.

This application is based upon and claims the benefit of priority fromJapanese patent application No. 2014-191480, filed on Sep. 19, 2014 andJapanese patent application No. 2014-257137, filed on Dec. 19, 2014, thedisclosures of which are incorporated herein in their entirety byreference.

1. An image processing device comprising: one or more memories storinginstructions; and one or more processors configured to execute theinstructions to: accept inputting of an operation for movement of atarget object on a captured image, the captured image including at leastone of a plurality of persons and a plurality of objects captured by acamera, the target object being indicated by a first image that issuperimposed on the captured image based on specific camera parametersrelating to a position and an attitude of the camera set on a realspace; and present, based on the specific camera parameters, atrajectory of the target object on the captured image, the trajectorycorresponding to the movement of the target object on the captured imageand passing through at least one position on the captured image, the atleast one position relating to the at least one of the plurality ofpersons and the plurality of objects, wherein the one or more processorsare further configured to: accept the operation for movement byrepeatedly accepting a designation of a first position on the capturedimage; generate, when the first position is designated, based on thespecific camera parameters, a predetermined shape and a predeterminedsize on a real space of the target object, and a second position on thereal space relating to the first position, the first image indicatingthe target object on the captured image appearing in a camera determinedby the specific camera parameters when the target object is disposed inthe second position; present the generated first image in the firstposition on the captured image; accept inputting of changing an angle ofthe target object presented on the captured image; and determine adirection of rotating the target object in accordance with a user'soperation.
 2. The image processing device according to claim 1, whereinthe one or more processors are further configured to: acquire heightinformation of the second position, and calculate the second positionbased on the specific camera parameters, the first position, and theheight information of the second position.
 3. The image processingdevice according to claim 2, wherein the one or more processors arefurther configured to acquire information indicating a height of aground surface in a real space as the height information of the secondposition.
 4. The image processing device according to claim 2, whereinthe one or more processors are further configured to acquire pieces ofinformation of different heights for a plurality of areas on thecaptured image, respectively, as the height information of the secondposition.
 5. The image processing device according to claim 1, whereinthe target object has a planar shape.
 6. The image processing deviceaccording to claim 1, wherein: the one or more processors are furtherconfigured to: accept inputting of a point or a line to the capturedimage; and display a second image indicating the point or the line uponmapping on a plane parallel to a ground surface, based on the specificcamera parameters, a position on the captured image of the point or theline, and height information on a real space of the point or the line.7. The image processing device according to claim 1, wherein the targetobject is a virtual cubic.
 8. The image processing device according toclaim 1, wherein the target object has a simplified shape and/or sizerepresenting the at least one of the plurality of persons and theplurality of objects.
 9. An image processing method comprising:accepting inputting of an operation for movement of a target object on acaptured image, the captured image including at least one of a pluralityof persons and a plurality of objects captured by a camera, the targetobject being indicated by a first image that is superimposed on thecaptured image based on specific camera parameters relating to aposition and an attitude of the camera set on a real space; presenting,based on the specific camera parameters, a trajectory of the targetobject on the captured image, the trajectory corresponding to themovement of the target object on the captured image and passing throughat least one position on the captured image, the at least one positionrelating to the at least one of the plurality of persons and theplurality of objects; accepting the operation for movement by repeatedlyaccepting a designation of a first position on the captured image;generating, when the first position is designated, based on the specificcamera parameters, a predetermined shape and a predetermined size on areal space of the target object, and a second position on the real spacerelating to the first position, the first image indicating the targetobject on the captured image appearing in a camera determined by thespecific camera parameters when the target object is disposed in thesecond position; presenting the generated first image in the firstposition on the captured image; accepting inputting of changing an angleof the target object presented on the captured image; and determining adirection of rotating the target object in accordance with a user'soperation.
 10. The image processing method according to claim 9, furtherincluding: acquiring height information of the second position, andcalculating the second position based on the specific camera parameters,the first position, and the height information of the second position.11. The image processing method according to claim 10, further includingacquiring information indicating a height of a ground surface in a realspace as the height information of the second position.
 12. The imageprocessing method according to claim 10, further including acquiringpieces of information of different heights for a plurality of areas onthe captured image, respectively, as the height information of thesecond position.
 13. The image processing method according to claim 9,wherein the target object has a planar shape.
 14. The image processingmethod according to claim 9, further including: accepting inputting of apoint or a line to the captured image; and displaying a second imageindicating the point or the line upon mapping on a plane parallel to aground surface, based on the specific camera parameters, a position onthe captured image of the point or the line, and height information on areal space of the point or the line.
 15. The image processing methodaccording to claim 9, wherein the target object is a virtual cubic. 16.The image processing method according to claim 9, wherein the targetobject has a simplified shape and/or size representing the at least oneof the plurality of persons and the plurality of objects.
 17. A tangibleand non-transitory computer-readable storage medium having computerprogram instructions stored thereon that, when executed by one or moreprocessors included in an image processing device, cause the imageprocessing device to: accept inputting of an operation for movement of atarget object on a captured image, the captured image including at leastone of a plurality of persons and a plurality of objects captured by acamera, the target object being indicated by a first image that issuperimposed on the captured image based on specific camera parametersrelating to a position and an attitude of the camera set on a realspace; present, based on the specific camera parameters, a trajectory ofthe target object on the captured image, the trajectory corresponding tothe movement of the target object on the captured image and passingthrough at least one position on the captured image, the at least oneposition relating to the at least one of the plurality of persons andthe plurality of objects; accept the operation for movement byrepeatedly accepting a designation of a first position on the capturedimage; generate, when the first position is designated, based on thespecific camera parameters, a predetermined shape and a predeterminedsize on a real space of the target object, and a second position on thereal space relating to the first position, the first image indicatingthe target object on the captured image appearing in a camera determinedby the specific camera parameters when the target object is disposed inthe second position; present the generated first image in the firstposition on the captured image; accept inputting of changing an angle ofthe target object presented on the captured image; and determine adirection of rotating the target object in accordance with a user'soperation.